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Thermal  infrared  images  of  the  ocean  obtained  from  satellite  sensors  are  widely  used  for 
the  study  of  ocean  dynamics.  The  derivation  of  mesoscale  ocean  information  from  satellite 
data  depends  to  a  large  extent  on  the  correct  interpretation  of  infrared  oceanographic  images. 
The  difficulty  of  the  image  analysis  and  understanding  problem  for  oceanographic  images  is 
due  in  large  part  to  the  lack  of  precise  mathematical  descriptions  of  the  ocean  features, 
coupled  with  the  time  varying  nature  of  these  features  and  the  complication  that  the  view  of 
the  ocean  surface  is  typically  obscured  by  clouds,  sometimes  almost  completely.  Towards  this 
objective,  the  present  paper  describes  a  hybrid  technique  that  utilizes  a  nonlinear  probabilistic 
relaxation  method  and  an  expert  system  for  the  oceanographic  image  interpretation  problem. 
This  paper  highlights  the  advantages  of  using  the  contextual  information  in  the  feature 
labeling  algorithm.  The  need  for  an  expert  system  and  its  feedback  in  automatic  interpretation 
of  oceanic  features  is  discussed.  The  paper  presents  some  important  results  of  the  series  of 
experiments  conducted  at  the  Remote  Sensing  Branch,  of  the  Naval  Oceanographic  and 
Atmospheric  Research  Laboratory,  on  the  National  Oceanic  and  Atmospheric  Administration 
Advanced  Very  High  Resolution  Radiometer  (AVHRR)  imagery  data.  The  results  clearly 
indicate  the  drastic  improvement  in  labeling  due  to  the  oceanographic  expert  system. 


Keywords:  Expert  system;  Knowledge-based  system;  Feature  labeling;  Oceanic  features; 
Relaxation;  Infrared  imagery. 

I.  INTRODUCTION 

Satellite-borne  sensors  potentially  offer  many  advantages  for  the  study  of  oceanic 
processes.  They  provide  global  synoptic  measurements  of  various  oceanic  surface 
properties,  in  contrast  to  the  local  measurements,  possibly  at  a  range  of  depths, 
provided  by  conventional  oceanographic  measurement  techniques.  Thermal  in¬ 
frared  (IR)  images  of  the  ocean  obtained  from  satellite  sensors  are  widely  used  for  the 
study  of  ocean  dynamics.  Figure  l  shows  a  sample  infrared  image  of  the  Gulf  Stream 
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Fig.  I.  Typical  IR  image. 


obtained  from  the  Advanced  Very  High  Resolution  Radiometer  (AVHRR)  aboard  the 
NOAA-7  satellite.  Brightness  in  this  infrared  image  is  inversely  proportional  to  the 
ocean  surface  temperature  (dark  areas  represent  warmer  temperatures  and  light  areas 
represent  colder  temperatures).  Vortices  (areas  of  closed  circulation)  within  this 
turbulent  flow  pattern  are  called  eddies.  The  Gulf  Stream  and  its  associated  eddies  are 
examples  of  mesoscale  features  (“mesoscale”  is  the  name  commonly  applied  to  the 
features  existing  on  spatial  scales  of  the  order  of  50  to  300  km).  Mesoscale  features 
are  important  to  the  study  of  ocean  dynamics,  to  fisheries  and  to  many  other  diverse 
interests. 

Current  image  analysis  techniques  rely  on  human  interpretation  of  the  satellite 
imagery.  Human  interpretation  is  obviously  varied  in  its  level  of  expertise  and  is 
highly  labor-intensive.  With  the  proliferation  of  high  volume  AVHRR  image  applica¬ 
tions,  it  becomes  highly  desirable  for  certain  applications  to  move  from  the  labor- 
intensive  manual  interpretation  of  infrared  imagery  towards  the  capability  for  auto¬ 
mated  interpretation  of  these  images.  The  complete  automation  of  the  oceanographic 
image  interpretation  function  is  probably  not  feasible,  but  one  can  begin  to  address 
certain  subsets  of  the  problem  with  the  present-day  image  processing  and  artificial 
intelligence  techniques.  This  was  the  motivation  for  the  work  reported  by  Lybanon 
et  al. , 1  in  the  development  of  a  prototype  oceanographic  expert  system. 
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Several  previous  studies  have  addressed  the  automation  of  the  analysis  of  inhered 
satellite  imagery  for  mesoscale  features.  Gerson  and  Gaborski,2  and  Gerson  et  al. 3 
investigated  the  detection  of  the  Gulf  Stream  in  infrared  images  from  the  Geostationary 
Operational  Environmental  Satellite  (GOES).  Gerson  and  Gaborski2  used  a  hierarchical 
approach  where  16  x  16  pixel  (128  x  128  km)  “frames”  within  the  image  were 
evaluated  for  the  possibility  of  containing  the  Gulf  Stream.  Frames  flagged  as  Gulf 
Stream  possibilities  were  then  further  evaluated  to  determine  the  exact  location  of  the 
Stream  within  the  frame  by  looking  at  statistics  based  on  5x5  pixel  “local 
neighborhoods”.  As  an  outgrowth  of  the  work  reported  in  Refs.  2  and  3,  Coulter4 
performed  automated  feature  extraction  studies  using  the  higher  resolution  Advanced 
Very  High  Resolution  Radiometer  data.  Janowitz5  studied  the  automatic  detection  of 
the  Gulf  Stream  eddies  using  Advanced  Very  High  Resolution  Radiometer  data. 
Nichol6  used  a  region  adjacency  graph  to  define  spatial  relationships  between 
elementary  connected  regions  of  constant  gray  level  called  atoms.  Although  satisfac¬ 
tory  emulation  of  human  extraction  of  eddy  structure  is  claimed  for  this  method, 
Nichol6  did  point  out  that  not  all  enclosed  uniform  areas  identified  by  the  method  will 
correspond  to  real  ocean  structure.  Krishnakumar  et  al.1  had  earlier  approached  this 
problem  and  shown  some  preliminary  results  on  the  relaxation-based  feature  labeling 
scheme,  the  development  of  a  prototype  expert  system  and  the  interface  between  these 
two  modules.  Goodenough  et  al*  have  developed  an  expert  system  for  remote  sensing 
applications.  In  Ref.  8  the  knowledge  is  presented  in  the  production  rules  and  frame 
databases.  The  expert  system  given  in  Ref.  8  is  used  to  study  the  spatial  differences 
between  maps  and  images. 

2.  MOTIVATION  OF  THE  PRESENT  WORK 

Our  primary  objective  is  to  build  a  powerful  automatic  image  interpretation  system 
for  oceanographic  satellite  images.  In  order  to  make  this  difficult  problem  tractable, 
we  divide  the  problem  into  two  parts:  the  feature  labeling  problem  and  the 
development  of  an  expert  system.  It  is  clear  that  the  performance  of  the  labeling 
algorithm  depends  heavily  on  the  low  level  image  processing  algorithms.  Particularly, 
the  output  of  an  edge  detector  algorithm  plays  a  major  role  in  the  feature  labeling 
process.  In  view  of  this,  a  new  efficient  edge  detector  algorithm  proposed  by  Holyer 
and  Peckinpaugh9  is  employed  in  our  feature  labeling  technique.  It  is  known  that  the 
conventional  edge  derivative  operators  are  very  sensitive  to  noise  and  are  not  suitable 
for  analyzing  oceanographic  satellite  images.  The  new  edge  detector  algorithm 
proposed  by  Holyer  and  Peckinpaugh9  is  based  on  the  gray  level  co-occurrence  matrix, 
which  is  commonly  used  in  image  texture  analysis.  This  algorithm9  is  found  to  exhibit 
the  characteristics  of  fine  structure  rejection  while  retaining  edge  sharpness.  In  this 
paper  we  focus  on  the  importance  of  an  expert  system  in  building  an  automatic 
oceanographic  image  interpretation  system. 

The  objective  of  an  oceanographic  expert  system  is  to  correctly  interpret  the 
dynamics  of  the  ocean  process  with  minimal  human  interaction.  Towards  this 
objective,  the  development  of  a  powerful  oceanographic  expert  system  is  under  way  at 
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the  Remote  Sensing  Branch  of  the  Naval  Oceanographic  and  Atmospheric  Research 
Laboratory  (NOARL).  However,  we  have  already  developed  a  prototype  expert  system 
for  Gulf  Stream  regional  dynamics.1  One  of  the  elements  of  an  expert  system  is  a 
database  of  knowledge  about  the  subject  matter,  with  the  knowledge  represented  in  a 
form  suitable  for  manipulation  by  the  “inference  engine’’  of  the  expert  system  (i.e. 
the  logical  and  heuristic  procedures  for  solving  problems  in  the  problem  domain). 
With  the  help  of  the  knowledge  gained  from  discussions  at  NOARL  and  from  the 
literature  in  oceanography,  a  knowledge  base  about  the  nature  of  the  mesoscale 
features  (occurrence,  mean  lifetime,  movement,  etc.)  was  built.1 

This  paper  describes  a  hybrid  technique  to  label  features  in  thermal  infrared  images 
obtained  from  satellites.  The  technique  presented  here  exploits  the  advantages  of  using 
contextual  information  in  the  labeling  algorithm.  The  remainder  of  the  paper  is 
organized  as  follows:  Section  3  gives  an  overview  of  the  architecture  of  the  proposed 
image  interpretation  system.  Section  4  introduces  the  probabilistic  relaxation  scheme 
and  briefly  discusses  the  application  of  this  scheme  to  the  oceanographic  labeling 
problem.  In  Sec.  4.1  we  develop  a  mathematical  framework  which  is  necessary  for  our 
labeling  problem.  Section  4.2  discusses  the  performance  of  our  technique  and 
describes  the  steps  that  are  involved  in  implementing  this  technique.  Section  S 
describes  the  need  for  an  oceanographic  expert  system  and  the  implementation  of  the 
expert  system.  The  improvement  in  feature  labeling  due  to  the  feedback  from  the 
expert  system  is  illustrated  in  Sec.  6  with  the  help  of  an  example.  Section  7  concludes 
with  all  the  important  features  of  this  technique  and  proposes  future  extensions  to  the 
available  scheme. 


3.  ARCHITECTURE  OF  THE  INTERPRETER 

In  this  section,  we  briefly  describe  the  various  components  involved  in  the  image 
iteratively  updated  using  the  nonlinear  probabilistic  relaxation  scheme.  Finally,  the 
architecture  of  the  proposed  system.  The  IR  imag;  obtained  from  the  satellite  is  the 
input  to  the  feature  labeling  module.  The  feature  labeling  module  consists  of  two 
submodules:  one  estimates  the  initial  feature  probabilities  and  the  other  implements  the 
iterative  updating  scheme.  The  initial  pixel  probabilities  are  assigned  with  the  help  of 
the  previous  analysis  and  the  ground  truth  data.  These  probability  values  are  the., 
iteratively  updated  using  the  nonlinear  probabilistic  relaxation  scheme.  Finally,  the 
relevant  features  that  are  present  in  the  image  are  labeled  with  the  aid  of  a  decision 
process. 

The  labeled  features  are  then  fed  to  the  expert  system  module.  This  expert  system  is 
divided  into  two  submodules.  The  first  submodule  consists  of  a  knowledge  base  and 
an  inference  engine.  The  knowledge  base  has  a  thorough  collection  of  facts  and  rules 
about  the  mesoscale  features  such  as  Gulf  Stream,  warm  core  rings,  etc.  Based  on  the 
positional  information  obtained  from  the  feature  labeling  module  and  the  knowledge 
base,  the  inference  engine  attempts  to  interpret  the  dynamics  of  these  features. 
Basically  the  inference  engine  is  a  pattern  matching  module.  The  second  submodule 
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Fig.  2.  Architecture  of  the  hybrid  system. 


evaluates  the  consistency  of  the  labeling  process  with  the  help  of  the  knowledge  base 
as  well  as  from  the  input  from  the  operator.  This  evaluation  results  in  assigning  a 
confidence  factor  to  each  of  the  features  detected.  This  factor  is  them  “fed  back”  to 
the  feature  labeling  module,  in  addition  to  the  previous  data  analysis,  to  improve  the 
consistency  of  the  labeling  process  in  future. 

A  salient  feature  of  this  hybrid  architecture  is  the  feedback  from  the  knowledge- 
based  system  to  the  labeling  module.  Due  to  atmospheric  effects  like  cloud  cover,  the 
oceanic  features  may  not  be  identified  by  the  feature  labeling  module.  Thus,  the 
position  of  the  Gulf  Stream  and  eddies  may  not  be  determined  correctly.  In  such 
cases,  the  expert  system  provides  an  approximate  position  of  these  features.  This  helps 
in  interpreting  the  subsequent  images.  This  is  illustrated  with  the  help  of  an  example 
in  Sec.  6.  Another  feature  is  the  dynamic  interpretation  of  the  features.  The  hybrid 
architecture  allows  the  expert  system  to  learn  from  previous  analysis.  Also  the  modular 
approach  allows  the  user  to  maintain  the  system  and  incorporate  changes  without  much 
difficulty. 
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4.  PROBABILISTIC  RELAXATION  LABELING 

An  important  research  area  in  image  analysis  and  image  interpretation  technology  is 
the  development  of  methods  that  blend  contextual  information  with  conventional  image 
processing  algorithms.  A  literature  survey  clearly  indicates  that  such  a  hybrid  approach 
yields  good  results.  Relaxation  labeling  is  one  such  process.  Relaxation  labeling  has 
been  applied  to  a  variety  of  image  processing  problems,  e.g.  linear  feature 
enhancement,10  edge  enhancement,11  image  enhancement,12  pixel  classification.'314  A 
recent  survey  article  by  Kittler  and  Illingworth15  on  relaxation  labeling  highlights  the 
importance  of  this  area  of  research.  The  survey15  also  points  out  the  advantages  and 
possible  applications  of  relaxation  methods.  More  importantly,  the  relaxation  labeling 
approach  was  elegantly  described  by  Rosenfeld  et  al. 16  who  investigated  the  problem 
of  labeling  the  sides  of  a  triangle  and  proposed  a  set  of  schemes  to  solve  the  problem. 
The  paper16  concluded  with  the  result  that  the  nonlinear  probabilistic  relaxation 
schemes  yield  better  results  than  the  others.  The  labeling  algorithm  presented  in  this 
paper  is  based  on  the  nonlinear  probabilistic  relaxation  technique. 

The  goal  of  the  relaxation  process  is  to  reduce  the  uncertainty  (and  improve  the 
consistency)  in  the  assignment  of  one  of  the  labels  to  each  object  in  a  set  of  related 
objects.  In  the  oceanographic  feature  classification  problem,  the  classes  are  the  various 
oceanographic  features,  namely  the  north  and  south  walls  of  the  Gulf  Stream,  cold 
eddy,  warm  eddy,  shelf  front  and  coastal  boundary.  Refer  to  Fig.  3  to  identify  the 
positions  of  these  oceanic  features  in  a  typical  image.  The  objects  are  the  individual 
pixels  in  a  set  of  registered  multi-temporal  images.  The  uncertainty  could  be  due  to 
cloud  cover  or  the  overlap  of  features,  features  not  belonging  to  one  of  the  classes, 
noise  in  the  image,  or  other  factors.  In  this  paper,  wc  are  attempting  to  label 
mesoscale  features,  but  the  ocean  exhibits  variability  on  all  spatial  scales.  Thermal 
structure  on  scales  smaller  than  mesoscale  will  interfere  with  the  mesoscale  feature 
labeling  process.  The  underlying  mathematical  framework  necessary  for  the  relaxation 
labeling  method  is  described  in  the  next  section. 

4.1.  Mathematical  Framework  For  Relaxation  Labeling 

Let  A  =  {A|,  A2,  .  .  .  ,  Am}  be  the  set  of  possible  labels  that  may  be  assigned  to 
each  pixel  x  in  the  IR  image.  Also  we  let  pk(x)  denote  the  probability  that  the  pixel  at 
x(i,  j)  belongs  to  the  object  A  after  k  iterations  of  the  relaxation  algorithm.  Note  that 
the  probabilities  are  functions  of  time  unlike  the  conventional  pixel  relaxation  labeling 
schemes  where  the  probability  is  a  function  of  position  alone.  This  allows  the 
relaxation  labeling  algorithm  to  utilize  temporal  continuity  to  reduce  the  ambiguity  in 
labeling.  The  ambiguity  may  arise  due  to  noise  (cloud  cover,  for  example). 

There  are  two  steps  in  executing  the  probabilistic  relaxation  algorithm.  In  the  first 
step,  a  priori  probabilities  are  evaluated  with  the  help  of  ground  truth  data  and/or  a 
previous  but  recent  mesoscale  analysis.  In  the  second  step,  these  a  priori  probabilities 
are  iteratively  updated  (relaxation)  until  a  consistent  labeling  is  reached.  We  now 
discuss  these  two  steps  in  detail. 
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Step  I.  Estimating  the  a  priori  probabilities. 

Let  p“(jc)  denote  the  a  priori  value,  that  is,  the  probability  that  pixel  x(i,  j)  belongs 
to  the  object  A  at  the  zeroth  iteration.  The  Bayesian  probability  equation  is  used  to 
evaluate  this  value.  Equation  (4. 1. 1)  is  used  to  calculate  /?A(jr). 


P»  = 


p{x |A)  P( A) 
5>(*|A)  P( A) 


(4.1.1) 


where  p(x | A)  denotes  the  conditional  density  function  and  P( A)  the  probability  of 
occurrence  of  the  object  A. 

To  evaluate  the  conditional  density  function  />(jc|A),  a  set  of  parameters  is  measured 
at  the  pixel  x(i,  j).  Let  X  denote  the  parameter  vector.  The  following  parameters  are 
used  to  form  the  vector  X : 

(1)  the  vector  from  origin  to  pixel  x(i,  j)  (both  the  magnitude  and  direction). 

(2)  gray  scale  intensity  value  at  the  pixel  x(i,  j). 

(3)  the  edge  magnitude  (Sec.  4.2  presents  the  chosen  edge  operator  algorithm). 

For  each  object,  the  mean  vector  p.x  and  the  covariance  matrix  are  computed. 
Also  it  is  assumed  that  the  conditional  density  function  follows  a  normal  distribution. 
Hence  the  conditional  density  function  p(x\ A)  is  evaluated  using  Eq.  (4.1.2). 


p(jr|A)  =  (2tt|2)a|)  1/2  exp  --  (X  -  MA)'Sr'(X  -  px) 


(4.1.2) 


To  compute  P( A),  relative  areas  of  the  objects  are  considered.  The  number  of  pixels  in 
the  object  A  is  «A.  Then  P(A)  can  be  calculated  using  Eq.  (4.1.3), 


P(  A)  = 


fix 

2aWA  ' 


(4.1.3) 


Step  2.  Iterative  updating  algorithm. 

We  now  discuss  the  probability  updating  rule.  The  new  estimate  of  the  probability 
of  A  at  x(i,  j)  is  given  by  (4.1.4). 


*+,,  .  _  pjU)  (1  +  <?*(*)) 
?X  *  2a PxM  (1  +  <?‘W) 


(4.1.4) 


where  q*(x)  is  called  the  update  factor. 

The  updating  factor  for  the  estimate  px(x)  at  the  fcth  iteration  is  given  by 
Eq.  (4.1.5). 


**<*)  =  ~  2>2a''aa’(-*,  y)pUy) 

m 


(4.1.5) 


43 


348 


N  KRISHNAKUMAR  ET  AL. 


where  m  is  the  number  of  objects.  In  this  equation,  rAA-(x,  y)  denotes  compatibility 
coefficients.  These  coefficients  are  computed  as  in  Refs.  14  and  16.  According  to  the 
relaxation  scheme,  rAA>(x,  y)  is  a  measure  of  the  probabilistic  compatibility  between 
label  A  on  point  x  and  label  A'  on  point  y,  and  has  the  following  characteristics: 

(1)  If  A  on  x  frequently  co-occurs  with  A'  on  y,  then  rAA-(x,  y)  >  0,  and  if  they 
always  co-occur,  then  rAA<(x,  y)  =  1. 

(2)  If  A  on  x  rarely  co-occurs  with  A'  on  y,  then  rAA-(x,  y)  <  0,  and  if  they  never 
co-occur,  then  rAA-(x,  y)  =  -I. 

(3)  If  A  on  x  occurs  independently  of  A'  on  y,  then  rAA-( x,  y)  =  0. 

4.2.  Discussion  of  the  Technique 

The  implementation  of  the  above  mentioned  technique  is  carried  out  in  two  stages. 
Stage  l 

At  the  first  stage,  a  priori  probabilities  are  estimated  using  a  manually  prepared 
mesoscale  analysis  from  a  time  period  of  two  days  prior  to  the  test  image.  The  objects 
present  in  the  oceanographic  IR  image  are  identified  with  the  help  of  previous 
analysis.  A  set  of  pixels  on  the  north  and  south  wall  of  the  Gulf  Stream  are  provided 
by  the  previous  analysis.  The  center  pixel  and  radii  of  the  eddies  are  also  given.  From 
the  parameter  vector  X  for  a  pixel  x(i,  j),  the  mean  vector  /iA  is  calculated.  Also  for 
each  object  A,  the  covariance  matrix  ]£)A  is  computed.  Equation  (4. 1 .2)  is  used  to 
compute  the  conditional  density  function  p(x|A).  Finally  the  initial  probability  p°(x)  is 
computed  using  Eq.  (4.1.1). 

Stage  2 

In  the  second  stage,  the  iterative  updating  rule  is  implemented.  The  compatibility 
coefficients  are  calculated  using  the  initial  class  probability  values  obtained  in  the  first 
stage.  These  are  fixed  during  the  update  process.  As  a  concluding  step  in  the  second 
stage  of  the  implementation,  the  iterative  updating  algorithm  is  implemented  using 
Eq.  (4.1.4).  The  iterative  algorithm  terminates  when  (/>A+’(x)  -  />* (jt))  <  e,  where  e 
is  a  very  small  quantity. 

Now  we  discuss  the  features  of  the  edge  detection  algorithm  proposed  by  Holyer 
and  Peckinpaugh.9  The  motivation  behind  the  development  of  such  a  new  edge 
detector  algorithm  is  to  aid  the  analysis  of  oceanographic  satellite  images.  The  popular 
derivative-based  edge  operators,  viz.  Sobel’s  operator,  are  shown  to  be  too  sensitive  to 
edge  fine-structure  and  to  weak  gradients  to  be  useful  in  this  application.  The  edge 
algorithm  proposed  by  Holyer  and  Peckinpaugh9  is  based  on  the  cluster  shade  texture 
measure,  which  is  derived  from  the  gray  level  co-occurrence  (GLC)  matrix.  Holyer 
and  Peckinpaugh9  have  suggested  that  the  edge  detection  technique  based  on  the  GLC 
matrix  can  be  effectively  used  in  automated  detection  of  mesoscale  features.  The 
(i,  j)th  element  of  the  GLC  matrix,  /*(/,  yjAx,  Ay),  is  the  relative  frequency  with 
which  two  image  elements,  separated  by  distance  (Ax,  Ay),  occur  in  the  image,  one 
with  intensity  level  /  and  the  other  with  intensity  level  j.  The  elements  of  the  GLC 
matrix  can  be  combined  in  many  different  ways  to  give  a  single  numerical  value  that 
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would  be  a  measure  of  the  edges  present  in  the  image.  Hoiyer  and  Peckinpaugh9  have 
used  a  cluster  shade  function  which  is  found  to  be  very  effective  in  the  edge  detection 
process.  The  new  edge  algorithm  computes  the  cluster  shade  function  at  each  pixel. 
Then  the  edges  are  detected  by  finding  the  significant  zero  crossings  in  the  cluster 
shade  image.  The  advantages  of  this  new  edge  algorithm  over  the  conventional 
derivative-based  techniques  are  discussed  in  Ref.  9.  It  is  known  that  using  large 
windows  in  derivative-based  edge  detector  algorithms  results  in  poor  smoothing.  This 
problem  is  circumvented  in  the  new  algorithm.  Because  edges  are  detected  by  finding 
zero  crossings,'  precisely  positioned  lines  result,  even  if  the  GLC  matrix  is  calculated 
using  a  larger  window.  So,  the  desired  edge  detection  characteristic  of  retaining  sharp 
edges  while  eliminating  edge  detail  is  achieved  with  the  help  of  the  new  algorithm.  As 
ar.  input  to  our  feature  labeling  algorithm,  we  used  the  image  output  generated  by 
cluster  shade  algorithm,  with  a  window  size  of  16  X  16  pixels  and  zero  crossing 
threshold  of  50.  The  edge  magnitudes  obtained  from  this  new  edge  detector  algorithm 
are  used  as  an  input  to  the  feature  labeling  algorithm.  In  particular  the  edge 
magnitudes  are  used  to  evaluate  a  priori  probability  values. 

5.  OCEANOGRAPHIC  EXPERT  SYSTEM 

The  Remote  Sensing  Branch  of  NOARL  studies  and  develops  methods  to  exploit 
satellite  data  to  provide  oceanographic  information.  The  Navy  needs  to  provide  such 
information  to  the  Fleet,  so  NOARL  has  an  objective  of  transforming  research 
techniques  for  the  oceanographic  interpretation  of  satellite  data  into  operational 
procedures.  From  a  large  volume  of  satellite  data,  combined  with  non-satellite  data, 
the  shipboard  user  will  be  required  to  produce  operationally  useful  tactical  products. 
Dramatic  improvement  will  be  required  in  our  ability  to  automate  the  image  analysis 
process. 

Interactive  image  processing  is  a  powerful  tool  in  the  hands  of  an  expert  image 
interpreter.  However,  human  interpretation  is  both  varied  in  its  level  of  expertise  and 
is  labor-intensive.  Expert  judgement  is  required  for  oceanographic  image  interpretation 
because  there  is  little  or  no  standardization  in  either  image  enhancement  or  interactive 
techniques.  NOARL  has  conducted  research  in  the  use  of  concepts  from  the  field  of 
artificial  intelligence  to  develop  knowledge-based  “expert  systems”  to  automate  some 
aspects  of  the  oceanographic  image  interpretation  task.  That  work  is  an  attempt  to 
remove  human-labor-intensive  and  varying-skill-level  elements  from  the  interpretation 
of  oceanographic  image  and  other  satellite  data.  Lybanon  et  al. 1  describe  the 
motivation  and  some  of  the  history  of  NOARL  in  this  area. 

The  rule  base  of  the  expert  system  is  derived  from  an  unprecedented  compilation  of 
information,  and  the  expert  system’s  mode  of  operation  is  innovative.  The  rule  base 
represents  oceanographic  knowledge  about  the  evolution  of  mesoscale  ocean  features 
in  the  Gulf  Stream  region  of  the  north  Atlantic  Ocean.  The  rules  are  applied  in  such  a 
way  that  the  expert  system  describes  the  kinematics  of  that  evolution.  The  expert 
system  is  presently  implemented  in  a  combination  of  OPS83,  C,  and  FORTRAN 
running  on  a  VAX  8300.  The  expert  system  uses  the  rules  about  the  features  to  evolve 
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an  initial  “state”  to  a  later  time.  The  new,  hypothesized  state  can  then  be  used  as  a 
new  initial  condition  and  the  process  is  repeated.  This  process  can  be  usefully  carried 
out  for  several  steps.  This  mode  of  operation  is  different  from  expert  systems  which 
use  a  (sometimes  lengthy)  chain  of  reasoning  to  reach  a  single  conclusion,  such  as  a 
system  for  medical  diagnosis. 

The  domain  of  the  expert  system  is  the  Gulf  Stream  region  of  the  north  Atlantic 
Ocean.  The  expert  system  has  different  rules  for  ring  and  Gulf  Stream  behavior  in 
each  of  nine  geographical  regions.  Figure  3  shows  the  operating  regions  of  the  expert 
system.  Aside  from  the  lines  that  separate  them,  the  regions  are  bounded  only  by  land. 
A  ring’s  behavior  as  hypothesized  by  the  expert  system  depends  upon  which  region  its 
center  is  in  at  the  beginning  of  a  time  step.  Basically,  the  motion  has  a  region- 
dependent  velocity  vector.  However,  a  ring  that  is  closer  than  a  certain  critical 
distance  from  the  Gulf  Stream  undergoes  a  modification  of  the  basic  motion.  The  Gulf 
Stream  interaction  rules  are  also  region-dependent.  The  details  depend  upon  how  close 
the  ring  is  to  the  Gulf  Stream;  the  Gulf  Stream  interaction  may  result  in  a  deflection  or 
a  looping  motion  of  the  ring,  with  possible  coalescence  with  tht  Stream  in  some  cases. 
Ring  sizes  decrease  with  time.  The  rate  of  decrease  depends  on  the  region  and  on 
whether  there  is  Gulf  Stream  interaction.  A  ring  that  shrinks  below  a  certain  size 
disappears.  Gulf  Stream  motion  is  modeled  as  downstream  propagation  of  meanders, 
with  region-dependent  phase  velocities  and  amplitude  factors.  While  this  is  only  a 
first-order  model,  there  is  justification  in  the  literature  for  this  behavior.  Refer  to 
Ref.  I  for  a  complete  description  of  the  expert  system. 
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6.  HYBRID  FEATURE  LABELING  TECHNIQUE— AN  EXAMPLE 

In  this  section  we  present  the  results  of  our  feature  labeling  technique.  We  show 
that  the  expert  system  greatly  helps  in  labeling  the  features  when  the  analysis  of  the 
previous  sample  image  fails.  This  situation  arises  when  the  oceanic  features  are 
completely  covered  by  cloud.  In  the  following  we  describe  the  steps  involved  in  the 
hybrid  feature  labeling  technique. 

The  system  starts  up  with  the  human  interpretation  of  an  image  that  is  obtained 
two  or  three  days  before  the  current  image.  The  initial  probabilities  are  computed 
based  on  the  analysis  of  the  previous  image.  Nonlinear  probabilistic  relaxation 
technique  is  applied  to  label  the  features  in  the  current  image.  Figure  1  shows  the  IR 
image  with  all  the  features.  The  output  of  the  edge  detector  algorithm  is  given  in 
Fig.  4.  Figure  5  shows  the  IR  image  with  all  the  edge  features  overlapped.  The 
positional  details  of  the  features  are  then  given  to  the  expert  system.  Sometime  he  IR 
image  of  the  ocean  may  not  be  clear.  That  is,  the  features  are  covered  by  cloud.  To 
analyze  such  an  image,  the  feedback  from  the  expert  system  is  used.  The  position  of 
the  oceanic  features  is  obtained  by  activating  the  inference  engine  of  the  expert 
system.  This  approach  is  different  from  the  conventional  one,  in  which  the  information 
is  always  provided  by  the  low  level  module.  The  concept  of  getting  feedback  from  a 
high  level  module  (like  an  expert  system)  is  totally  new  and  justified  in  automatic 
image  interpretation  of  oceanographic  images.  The  output  of  the  labeling  algorithm  is 
given  in  Fig.  6. 


Fig.  4.  Output  of  the  edge  detector. 
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Fig.  5.  IR  (mage  with  edges  overlapped. 


Fig.  6.  Output  of  the  feature  labeling  algorithm. 
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7.  SUMMARY  AND  RECOMMENDATIONS  FOR  FUTURE  WORK 

In  this  paper,  the  need  for  automatic  interpretation  of  oceanographic  images  is 
emphasized.  The  advantage  of  exploiting  the  contextual  information  in  feature  labeling 
is  highlighted.  An  efficient  and  simple  technique  for  labeling  of  oceanic  features  is 
described.  The  underlying  theoretical  framework  and  the  steps  involved  in  estimating 
the  probability  functions  are  explained  in  detail.  The  overall  architecture  of  the 
proposed  image  interpretation  system  is  presented.  The  usefulness  of  an  expert  system 
in  feature  labeling  is  discussed.  The  concept  of  getting  feedback  from  the  expert 
system  is  a  new  approach  and  it  is  explained  with  the  help  of  an  example. 

As  a  future  extension  to  the  present  work  we  propose  to: 

(i)  make  the  oceanographic  expert  system  “learn”  from  its  past  experience  in 
analyzing  the  satellite  imagery  data. 

(ii)  investigate  the  possibility  of  implementing  a  parallel  relaxation  labeling  algorithm 
to  speed  up  the  labeling  process. 
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